TEST SITUATION, TEST CASE AND TEST SCRIPTS

In the ideal case, testing would give us the certainty that the system behaved as required or desired under all circumstances. In reality, not every circumstance can be tested – only a subset that is a direct result of the decisions and choices made during the test design phase.

The test design leads to a hierarchical structure of test situations – test cases – test scripts. These terms are discussed in detail further on in this section. The relationship between the concepts is shown in figure 1 and can be summarised as follows:

• Every test situation occurs in at least 1 test case
• A logical test case covers 1 or more test situations
• Every logical test case is worked out concretely into 1 physical test case
• Every physical test case occurs in 1 test script.

The figure 1 also shows the distinction between the logical and physical parts of the test design:

• The logical test design consists of the test situations and the logical test cases. This is the part that demonstrates that the required coverage is achieved, thereby complying with the test strategy.
• The physical test design consists of the concretely created physical test cases, laid down in test scripts. This guarantees a thorough preparation of the ‘Execution’ phase. The physical creation of test cases therefore adds nothing to the thoroughness of the test.

Figure 1: Relationships between the terms test situation – test case – test script.

"A test situation is an isolated condition under which the test object displays a specific behaviour that needs to be tested."

Example - ‘an order of more than 1 book’. Another test situation that is closely related would be: ‘an order of exactly 1 book’. And if, for instance, a discount on an order would not be offered until the order price exceeds a specific threshold value, the following test situations would be logical: ‘order price above threshold value’, and ‘order price below threshold value’.

A test case must therefore contain all of the ingredients to cause that system behaviour and determine whether or not it is correct. A well-known way to
describe system behaviour is ‘Input → Processing → Output’. To be able to test the system behaviour, a test case must match this perfectly and 'embrace’ it, as it were (see figure 2). Therefore the following elements must be recognisable in every test case regardless of the test design technique used:

• Initial situation - This covers everything that is needed to prepare the system for receiving the required input. This includes not only the data that are needed for the processing, but also the condition in which the system and its environment must be. For instance, one might think of setting a specific system date, or running specific week and month batches that bring the system to a specific status.
• Actions - This means all of the activities that must be executed to activate the system to the processing. It might be a simple command (‘Run …’) or entering specific data on a screen. But it can also be a complex sequence of entering parameters, activating a specific function, manipulating other data, starting up another function, etc.
• Predicted result - This covers all of the results that the tester must check to establish whether the system behaviour conforms to the expectations. Often, predicted result is incorrectly thought to be limited to the output that appears on screen or is stored in databases. But the system can also produce output that is transmitted to other systems or peripheral equipment. Furthermore, more than just output data may have to be checked to established that the system is working correctly. For instance:
  • ‘How quickly should the output appear?’
  • ‘What is the maximum allowed memory load and is it released afterwards?’
  • ‘Should the system produce interim signals or messages, such as the hourglass or beeps?’

Figure 2: Generic structure of a test case in relation to the system behaviour to be tested.

"A logical test case describes, in logical terms, the circumstances in which the system behaviour is examined by indicating which test situations are covered by the test case."

• order of more than 1 book (A1)
• order of exactly 1 book (A2)
• order price above threshold value (B1)
• order price below threshold value (B2)

These test situations are covered by, for instance, the following 2 logical test cases:

• TC-1: Order for more than 1 book, with the order price remaining below the threshold value.
• TC-2: Order for 1 book, with the order price above the threshold value.

Clearly this small, simple example is very trivial. And it can probably be described much more practically and compactly. (E.g. TC-1 = A1+B2) But the
point is that the logical test cases show, in a clearly structured manner:

• What the test situations are
• That the set of test cases covers all defined test situations.

Physical test cases

"A physical test case is the concrete elaboration of a logical test case, with choices having been made for the values of all required the input and settings of the environmental factors."

The physical test case describes in practical terms what has to be done. The 3 basic elements of a test case are recognisable:

• What needs to be prepared? (Initial situation)
• What does the tester have to do? (Action)
• What is the expected result? (Predicted result).

Threshold price = € 50.00

discount = 10%

price book-01 = € 18.50

price book-02 = € 25.50

price book-03 = € 65.00

TC-1:

Action: Order book-01 and book-02

Predicted result: No discount will be given. Order price = € 44.00

TC-2:

Action: Order book-03

Predicted result: A 10% discount will be given. Order price = € 58.50 (€ 65.00 - € 6.50 )

The step from logical to physical test case is more than just inserting concrete values. It is also a step from ‘theoretical’ to ‘practical’. In this step, the tester needs to look beyond the system specifications on which the logical test cases were based and ask himself: ‘Do I now know everything I need to execute this test case in the Execution phase?’ For instance:

• In the initial situation - Is the tester allowed to determine himself what data are entered in the test database? Or is use of existing test databases required?
• In the predicted result - How will it be established that the requirements are complied with? Is it sufficient that the datum appears on screen correctly, or must it (also) be retrieved from the database via a Read function?

Choices in the elaboration of the physical test design

The elaboration of the logical test design into a physical test design is an investment made in the Specification phase – outside the project’s critical path – to result in a saving in the Execution phase, which is on the critical path. The test organisation needs to decide:

• How far the tester must go in the physical elaboration of his test cases - Two extremes exist in this respect:
  • In the extreme situation, the test case is not worked out concretely, but remains described in abstract terms. It is left to the tester to work out the correct concrete content ‘on site’ when the test is executed.
  • In the other extreme, it is described to the lowest level of detail what the tester needs to do exactly: Which value should be where on the screen and which key then needs to be pressed.
• In what form the result must be registered - The most common ways to register test cases and test scripts are: text document, spreadsheet, database, in a test management tool. It might also be a mix. For example: the logical test design (test situations and logical test cases) is described in a text document while a spreadsheet elaborating the physical test cases is created for each individual test script.

Some considerations in the choice as to in how far the test cases need to be elaborated physically:

• Knowledge and experience of the testers - This mainly involves expertise on the subject matter and knowledge about how the automated system works. The less knowledge and experience, the more the tester needs to be told what to do exactly when executing the test.
• Time pressure in the Execution phase - The less the test cases are elaborated and prepared concretely in advance, the more ‘time to think’ the tester needs when executing the test (and vice versa). Normally, the Execution phase is on the critical path of the project. The higher the time pressure in that phase, the greater the necessity to have to do as little as possible in this phase – and therefore to have prepared the maximum properly.
• Transferability to other testers - A tester who has himself designed the test cases will require less comprehensive information to know exactly how to execute the test. However, if it must be possible for the test to be executed by people who were not involved in the creation of the test cases, the physical test cases and test scripts must contain more extensive information.

Some considerations when selecting in what form the result is to be registered:

• Availability of tools and knowledge about their use - This is a trivial point. It is no use prescribing the form in which the test cases and test scripts need to be registered if the testers do not have access to the tool required or do not know how to use it.
• Transferability to other environments - This is an important consideration if there are several environments on which the test products must be installed and used. Text documents are easily exported to other environments. Spreadsheets and databases are a bit more difficult. And transferring test products embedded in a test management tool often requires having an identical installation of exactly the same tool in the other environment.
• Interface with automated test execution - In case of far-reaching test automation, the tests are executed by an automated test suite that uses a specific test tool. This means that the physical test cases and test scripts must be recorded in a way that that tool can read. Often it is decided to register in spreadsheets, databases or the tool itself.

Test scripts

In a test script, multiple physical test cases are combined to be able to execute them in an efficient and simple manner.

There are no strict rules that say which test cases must be combined in a test script or how many test scripts are allowed. The tester has full freedom provided that the test scripts together contain all the test cases. The reasons to combine specific test cases into one test script are of a purely practical nature. For example:

• They use the same initial situation
• The same time-consuming actions must be executed for all test cases
• They all have to do with the same exceptional situation. For example: the test script contains all test cases about clients abroad
• There are timeline-dependent test cases. Before specific test cases can be executed, other test cases must have been completed first.

From a test-management perspective, test scripts are excellent, manageable units. Planned hours can be linked to each test script. Progress can be measured and reported in terms of test scripts.

COVERAGE, COVERAGE TYPE AND COVERAGE RATIO

There are a lot of definitions relating to coverage. The definition below is a simple one, but it reflects the essence perfectly well: "Coverage is the ratio between that which can be tested and that which is tested with the test set."

In other words, coverage says something about how many of ‘all possibilities that can be tested’ are actually tested. This means that there are 2  phenomena that together determine the coverage:

• Which possibilities are distinguished? This is termed coverage type. It indicates what type of possibilities are involved. These are subsequently elaborated into the required test situations. For instance, it is possible to look at the possible combinations of paths in an algorithm. But in that same algorithm, you can also look at the possibilities to cover one outcome or the other within each decision point. These are 2 different forms of coverage (i.e. ‘coverage types’ -"Coverage type is the form in which the covering of test situations deducible from the test basis, is expressed.)."
• How many of these possibilities are tested? This is called the coverage ratio and is generally expressed as a percentage. It shows which part of all of the alternatives have actually been tested. The number (percentage) does not have meaning until it is clear which possibilities are involved, i.e. when it is associated with the coverage type. "Coverage ratio is the percentage of test situations, as defined by the coverage type, that is covered by the test."

This section discusses the how and why of coverage, coverage type and coverage ratio in detail. It first discusses why coverage is a useful and practical concept for testers. It then explains, using a comprehensive example, why it is necessary to distinguish several types of coverage. Finally, it describes how test design techniques constitute the link between test cases and the coverage they achieve.

Why coverage?

‘Coverage’ is a useful concept that helps testers with complex questions, such as:

• Since it is impossible to test everything and we are therefore forced to test only a subset of all of the possibilities, what is the best subset?
• What is the difference between ‘elementary testing’ and ‘thorough testing’? What does it mean concretely for the test cases we need for this?

Coverage has everything to do with the wish to find the most possible defects with the fewest possible test cases. Instead of simply testing ‘just any’ subset of options, we aim to compile a set of test cases that create the greatest possible change of fi nding the defects that are there.

We can never be certain that all defects have been found or even that 60% of all defects has been found. After all, we do not know how many defects there actually are. What we can demonstrate, is the coverage realised by the test. And this confers a certain level of confidence that the chance of any defects remaining in the tested system is small. In summary: the higher the coverage realised, the smaller the chance that any unknown defects remain in the system.

The decision to ‘test more thoroughly’ concretely translates to a decision to achieve more extensive coverage. In principle, there are 3 options in this respect:

• A more thorough coverage type
• Multiple coverage types
• A higher coverage ratio of a specific coverage type.

One type of coverage or more?

It should be clear by now that a statement like ‘I want to test with a 75% coverage ratio’ does not say much, for the following reasons:

• What type of coverage are we talking about?
• Why does 25% (and which) not require coverage? (In other words: Why not simply cover 100%?)

This section will discuss these issues in greater detail and explain

• that we cannot talk about the coverage, but talk of many forms of coverage (i.e. ‘coverage types’)
• that the choice for more thorough testing must be worked out primarily with the choice for another coverage type
• that the use of a coverage type implies that one aims for a coverage ratio of 100%
• that generally speaking, the various coverage types CANNOT be compared in terms of ‘X is better than Y’. Different coverage types tend to supplement rather than replace each other.

To explain this, we will use an example One type of coverage or more about a system to order books via the Internet. It describes only a fraction of what such a system could do in reality, but it will suffice for our purposes.

So what is best? Do we choose covering paths or covering decision points? Clearly doing both achieves the highest level of thoroughness. But say that this is not allowed (e.g. because it is too costly). Say that we have to choose, then what would be best? Or more concretely: Which coverage type is better: ‘test depth level 2’ or ‘multiple condition coverage’? This question may seem simple, but it isn’t. One coverage type could only be called  better’ with any practical use if it would find at least all of the defects found by the other coverage type plus some additional defects. But the example above demonstrated that multiple condition coverage finds a specific defect that is not found with test depth level 2 (i.e. the extra discount for few but expensive books). But the reverse is true as well (i.e. the discount that is given incorrectly in case of cancellation). So the best answer is: We cannot unequivocally talk about ‘better’! The various coverage types find different defects.

If multiple coverage types can be used from which a choice has to be made, consult with the client. Analyse where the highest risks are located and select the coverage type that matches this area most adequately. Then make sure that the client is aware of the fact that there are risks in the area relating to the coverage type that was not selected.

For instance, the tester and client must typically ask themselves the following for the earlier question whether to choose test depth level 2 or multiple condition coverage: Is it a process in which many different types of actions are executed whose consequences might affect each other? In this case, path coverage test depth level 2 would be a good choice. But in case of a process where a specific discount is given or not after every decision point, path combinations are not that interesting because the discounts do not have a mutual impact. However, the client might consider the possible situations in which the discount is given or not given extremely important. In this case, coverage type multiple condition coverage might be the wiser choice.

TEST DESIGN TECHNIQUE AND BASIC TECHNIQUE

Techniques as a link between coverage and test cases

A test design technique is used to derive the necessary test cases that achieve the required coverage from a specific test basis. Such deriving from test cases is not a direct process – it occurs by determining the required coverage type(s) and deriving the test situations to achieve that coverage type (see figure 4). This is explained further below.

The required coverage is expressed concretely in the selected coverage types. Each coverage type requires a specific type of information in the test basis, e.g. a structured flow chart with paths and decision points. A standard method to derive the required test situations can be created for a specific coverage type. This is called a basic technique.

"Basic technique is the method of deriving test situations from the test basis to achieve the required coverage type."


Figure 4: Deriving test situations from coverage type.

Please note that for some basic techniques, a tool can be used to derive the required test situations.

How are techniques related to quality characteristics/test types?

The test design technique is closely related to the test type that is executed and thereby the quality characteristic that is tested with that test type. A test design technique describes the required test basis, the intended coverage types, and the basic techniques applied to achieve these coverage types.

A basic technique is NOT, in principle, related to a quality characteristic. For instance, path coverage with test depth level 2 (a coverage type) can be used to test functionality but also e.g. security, provided that the security specifications are described in terms of paths and decision points. To clarify the distinction and relationship between test design technique, basic technique and quality characteristic, consider the following two test design techniques:

• The Process Cycle Test is a test design technique to test the ‘suitability’ quality characteristic. The required test basis is the description of the AO (Administrative Organisation) in terms of paths and decision points. The intended coverage type is ‘path coverage test depth level 2’.
• The Algorithm Test (see http://www.tmap.net/) is a test design technique to test the ‘functionality’ quality characteristic. The test basis is the algorithm description of a piece of program code in terms of paths and decision points. The intended coverage type is ‘path coverage test depth level 2’.

These two test design techniques each have a different aim (a different quality characteristic is being tested) and work with completely different test objects (AO procedures and program code). But both test design techniques aim for the same coverage type (path coverage test depth level 2) and therefore use the same basic technique. In fact a tester only needs to learn one single basic technique (path coverage test depth level 2) to be able to use both test design techniques.